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5 TECHNICAL FIELD 

[0001] The present invention relates to a technology for 

representing code data by arranging cells in a two- 
dimensional array . 



10 BACKGROUND TECHNOLOGY 

[0002] Recently, the technology is widely available for 

capturing an image of a two-dimensional code by a camera for 
recognition and allowing a predetermined process associated 
with the code data to be performed. Compared to one- 

15 dimensional codes, two-dimensional codes can store a larger 
amount of coded information. Various kinds of two- 
dimensional codes are available these days. The data for a 
two-dimensional code needs to be read from image data 
efficiently and properly. In this background, technologies 

20 related to image recognition of two-dimensional codes have 
been proposed (e.g. patent document 1). 
patent document 1 

JP 2000-82108 

25 DISCLOSURE OF THE INVENTION 

[0003] If the resolution of image data is high, a large 
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amount of information for two-dimensional codes can be 
acquired so that a high recognition rate of code data is 
achieved. It is favorable to attain a high rate of 
recognition of code data even with a camera with relatively 
5 low resolution. Image data captured contains influence from, 
for example, ambient light outside a camera. In. this regard, 
it is favorable to properly configure a range of pixel values 
for binarization of the image data captured, by allowing for 
the influence. 

10 A general purpose of the present invention in this 

background is to provide a technology capable of recognizing 
a two-dimensional code or an article such as a card with high 
precision . 

[0004] An article according to at least one embodiment 

15 of the present invention comprises: a code data field 
including a two-dimensional array of a plurality of 
rectangular cells to form code data; and a plurality of 
corner cells provided to surround the code data field, 
wherein at least one of the corner cells occupies a larger 
20 area than at least one of the rectangular cells. The article 
may be a two-dimensional object such as a card or a three- 
dimensional object. The rectangular cells and the corner 
cells may be printed on the article or engraved on the 
article. An essential requirement is that the cells are 
25 provided such that images thereof can be captured by the 
imaging apparatus . 
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[0005] An article according to at least one embodiment 

of the present invention comprises: a code data field 
including a two-dimensional array of a plurality of 
rectangular cells to form code data; and a plurality of 
5 corner cells provided to surround the code data field, 

wherein the corner cells are round in shape. The article may 
be a two-dimensional object such as a card or a three- 
dimensional object. The rectangular cells and the corner 
cells may be printed on the article or engraved on the 
10 article. An essential requirement is that the cells are 
provided such that images thereof can be captured by the 
imaging apparatus . 

[0006] There is also provided image data generated in 

accordance with a data format for displaying a two- 

15 dimensional code on a display, wherein the image data 

comprises: data for displaying a plurality of rectangular 
• cells at predetermined coordinate positions on the display; 
and data for displaying a plurality of corner cells at 
coordinate positions surrounding the plurality of rectangular 

20 cells, wherein the data for the corner cells and the 

rectangular cells is configured such that at least one of the 
corner cells appears larger than at least one of the 
rectangular cells when displayed. 

[0007] There is also provided image data generated in 

25 accordance with a data format for displaying a two- 
dimensional code on a display, wherein the image data 
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comprises: data for displaying a plurality of rectangular 
cells at predetermined coordinate positions on the display; 
and data for displaying a plurality of corner cells at 
coordinate positions surrounding the plurality of rectangular 
5 cells which are displayed at the predetermined coordinate 
positions, wherein the data for the corner cells is 
configured such that the corner cells appear round when 
displayed. 

[0008] In a method according to at least one embodiment 

10 of the present invention for transmitting image data 

generated in accordance with a data format for displaying a 
two-dimensional code on a display, the image data comprises: 
data for displaying a plurality of rectangular cells at 
predetermined coordinate positions on the display, data for 
15 displaying a plurality of corner cells at coordinate 

positions surrounding the plurality of rectangular cells 
which are displayed at the predetermined coordinate positions, 
and the data for the corner cells and the rectangular cells 
is configured such that at least one of the corner cells 
20 appears larger than at least one of the rectangular cells 
when displayed. 

[0009] In a method according to at least one embodiment 

of the present invention for transmitting image data 
generated in accordance with a data format for displaying a 
25 two-dimensional code on a display, the image data transmitted 
comprises: data for displaying a plurality of rectangular 
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cells at predetermined coordinate positions on the display; 
data for displaying a plurality of corner cells at coordinate 
positions surrounding the plurality of rectangular cells, 
wherein the data for the corner cells is configured such that 
5 at least one of the corner cells appears round when displayed. 
[0010] An article according to at least one embodiment 

of the present invention comprises: a reference cell having a 
predetermined shape; and a plurality of polygonal cells in a 
two-dimensional array to form code data; and a plurality of 

10 corner cells provided to surround an area with the two- 
dimensional array of the plurality of polygonal cells , 
wherein at least one of the reference cell, the polygonal 
cells and the corner cells is assigned a color different from 
that of the other cells. The article may be a two- 

15 dimensional object such as a card or a three-dimensional 

object. The rectangular cells and the corner cells may be 
printed on the article or engraved on the article. An 
essential requirement iis that the cells are provided such 
that images thereof can be captured by the imaging apparatus. 

20 [0011] There is also provided image data generated in 

accordance with a data format for displaying a two- 
dimensional code on a display, wherein the image data 
transmitted comprises: data for displaying a reference cell 
having a predetermined shape at a predetermined position on 

25 the display; data for displaying a plurality of polygonal 
cells within an area defined with respect to the reference 
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cell; and data for displaying a plurality of corner cells at 
coordinate positions surrounding the area, wherein at least 
one of the reference cell, the polygonal cells and the corner 
cells is assigned a color different from that of the other 
5 cells. 

[0012] In a method according to at least one embodiment 

of the present invention for transmitting image data 
generated in accordance with a data format for displaying a 
two-dimensional code on a display, the image data transmitted 

10 comprises: data for displaying a reference cell having a 
predetermined shape at a predetermined position on the 
display, data for displaying a plurality of polygonal cells 
within an area defined with respect to the reference cell; 
and data for displaying a plurality of corner cells at 

15 coordinate positions surrounding the area, wherein at least 

one of the reference cell, the polygonal cells and the corner 
cells is assigned a color different from that of the other 
cells . 

[0013] An image analysis apparatus for recognizing an 

20 object image included in a frame image is also provided. 
The object image includes a reference cell having a 
predetermined shape and a plurality of corner cells assigned 
a color different from that of the reference cell. The image 
analysis apparatus comprises: a binarization processor which 
25 sets up a range of pixel values in the RGB format so as to 

translate the frame image into a binary bit representation; a 
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first detector which detects the reference cell in binary 
data for the frame image; and a second detector which detects 
the corner cells in the binary data for the frame image. The 
binarization processor sets up a range of pixel values in the 
5 RGB format for detecting the reference cell and binarizes the 
frame image by extracting pixels within the range thus set up, 
whereupon the first detector detects the reference cell by 
referring to the resultant binary data. The binarization 
processor sets up a range of pixel values in the RGB format 
10 for detecting the corner cells and binarizes the frame data 
by extracting pixels within the range thus set up, whereupon 
the second detector detects the corner cells by referring to 
the resultant binary data. 

[0014] A method for recognizing an object image included 

15 in a frame image is also provided. The object image includes 
a reference cell having a predetermined shape and a plurality 
of corner cells assigned a color different from that of the 
reference cell. The method comprises": setting up a range of 
pixel values in the RGB format for detecting the reference 
20 cell, extracting pixels within the range thus set up, and 

detecting the reference cell in the frame image; and setting 
up a range of pixel values in the RGB format for detecting 
the corner cells, extracting pixels within the range thus set 
up, and detecting the corner cells in the frame image. 
25 [0015] An article according to at least one embodiment 

of the present invention comprises: a reference cell having a 
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predetermined shape; and a plurality of polygonal cells in a 
two-dimensional array to form code data; and a plurality of 
corner cells having an identical shape. The corner cells are 
assigned a color different from that of the reference cell 
5 and the polygonal cells. The article may be a two- 
dimensional object such as a card or a three-dimensional 
object. 

[0016] A game mat is also provided on which a game card 

is placed and which is provided with a gradation area in 

10 which brightness varies gradually. 

[0017] A card game system according to at least one 

embodiment of the present invention comprises: a game card 
having a cell assigned a predetermined color; a game mat 
assigned a color of the same hue as the cell of the game card 

15 and provided with a gradation area in which brightness varies 
gradually; an imager which acquires a frame image by 
capturing an image of the game mat and an image of the game 
card placed on the game mat; a binarization processor which 
sets up a range of pixel values in the RGB format and 

20 translates the frame image into a binary bit representation; 
an extractor which extracts the gradation area on the game 
mat, from binary data for the frame image; and an adjuster 
which refers to binary data for the gradation area and 
accordingly adjusts the range of pixel values in the RGB 

25 format for binarization by the binarization processor. 

[0018] An image analysis apparatus according to at least 
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one embodiment of the present invention for analyzing a frame 
image capturing an image of a game card having a cell 
assigned a predetermined color and an image of a game mat 
which is assigned a color of the same hue as the cell and 
5 which is provided with a gradation area in which brightness 
varies gradually , comprises: a binarization processor which 
sets up a range of pixel values in the RGB format so as to 
translate the frame image into a binary bit representation; 
an extractor which extracts the gradation area on the game 

10 mat, from binary data for the frame image; and an adjuster 
which refers to binary data for the gradation area and 
accordingly adjusts the range of pixel values in the RGB 
format for binarization by the binarization processor. 
[0019] An image analysis method according to at least 

15 one embodiment of the present invention for analyzing a frame 
image capturing an image of a game card having a cell 
assigned a predetermined color and an image of a game mat 
which is assigned a color of the same hue as the cell of the 
game card and which is provided with a gradation area in 

20 which brightness varies gradually, comprises: setting up a 
range of pixel values in the RGB format so as to translate 
the frame image into a binary bit representation; extracting 
the gradation area on the game mat, from binary data for the 
frame image; and referring to binary data for the gradation 

25 area and accordingly adjusts the range of pixel values in the 
RGB format for binarization. 
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[0020] A program product according to at least one 

embodiment of the present invention comprising computer 
readable codes for causing a computer to analyze a frame 
image that captures an image of a game card having a cell 
5 assigned a predetermined color and an image of a game mat 
which is assigned a color of the same hue as the cell and 
which is provided with a gradation area in which brightness 
varies gradually, comprises: a translating module which sets 
up a range of pixel values in the RGB format so as to 

10 translate the frame image into a binary bit representation; 

an extracting module which extracts the gradation area on the 
game mat, from binary data for the frame image; and an 
adjusting module which refers to binary data for the 
gradation area and accordingly adjusts the range of pixel 

15 values in the RGB format for binarization . 

[0021] Optional combinations of the aforementioned 

constituting elements, and implementations of the invention 
in the form of methods, apparatuses, systems, recording 
mediums and computer programs may also be practiced as 

20 additional modes of the present invention. 

[0022] According to the present invention, a two- 

dimensional code that can be recognized with high precision 
is provided. 

25 BRIEF DESCRIPTION OF THE DRAWINGS 
[0023] 
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Fig. 1 shows the structure of a game system according 
one example of the invention; 

Fig. 2 shows a two-dimensional code printed on the 
surface of a game card; 
5 Fig. 3 shows the structure of an image analysis 

apparatus; 

Fig. 4 shows a processing procedure for recognition of 
two-dimensional codes; 

Fig. 5 is a flowchart showing the process of detecting 
10 a reference cell; 

Fig. 6 is a flowchart showing the process of detecting 
a code field; 

Fig. 7 is a flowchart showing the process of verifying 
code data; 

15 Fig. 8 shows another example of two-dimensional code 

printed on the surface of a game card; 

Fig. 9 shows still another example of two-dimensional 
code printed on the surface of a game card; 

Fig. 10 shows yet another example of two-dimensional 
20 code printed on the surface of a game card; 

Figs. 11A-11C are charts showing a method of 
determining whether a corner cell is found; 

Fig. 12 shows another example of two-dimensional code 
printed on the surface of a game card; 
25 Fig. 13 shows a game mat on which to place a game card; 

Fig. 14 shows the appearance of a gradation area; 
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Fig. 15 shows another structure of an image analysis 
apparatus; 

Figs. 16A-16C show doughnut areas identified as 
gradation areas; and 
5 Fig. 17 shows a system of transmitting image data 

according to the example. 
[0024] 

1 ... game system, 2 ... imaging apparatus, 4 ... game card, 
6 ... output apparatus, 7 ... display, 10 ... image 

10 processing apparatus, 20 ... image analysis apparatus, 30 
game apparatus, 4 0 ... frame image acquirer, 42 ... real 
object extractor, 44 state determiner, 50 ... binarization 
processor, 52 ... reference cell detector, 54 ... code field 
detector, 56 ... verification processor, 60 ... area 

15 extractor, 62 ... RGB adjuster, 100 . . . reference cell, 

102 . . . code data field, 104 corner cell, 106 rectangular 
cell, 108 . . . area, 110 . . . code field, 200 . . . image 
transmission system, 201 . . . image data generation apparatus, 
202 . . . terminal, 203 display apparatus, 204 . . . transmission 

20 station 

BEST MODE FOR CARRYING OUT THE INVENTION 

[0025] Fig. 1 shows the structure of a game system 1 

according to an example of the present invention. The game 
25 system 1 is provided with an imaging apparatus 2, an image 

processing apparatus 10 and an output apparatus 6. The image 
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processing apparatus 10 is provided with an image analysis 
apparatus 20 and a game apparatus 30. The image analysis 
apparatus 20 and the game apparatus 30 may be separate 
apparatuses or may be integrally combined- The imaging 
5 apparatus 2 is embodied by a video camera comprising a charge 
coupled device (CCD) imaging element, a metal oxide 
semiconductor (MOS) imaging element or the like. For example, 
the imaging apparatus 2 has a resolution of 320 by 240 pixels. 
The imaging apparatus 2 captures an image of a real space 

10 periodically so as to generate a frame image in each period. 
An imaging area 5 represents a range captured by the imaging 
apparatus 2. The position and size of the imaging area 5 are 
adjusted by adjusting the height and orientation of the 
imaging apparatus 2. A game player manipulates a game card 4, 

15 a real object, in the imaging area 5. The game card 4 is 

provided with a two-dimensional code that uniquely identifies 
the card. 

[0026] The output apparatus 6 is provided with a display 

7. The output apparatus 6 may also be provided with a 

20 speaker (not shown) . The image processing apparatus 10 

causes the display 7 to display a frame image captured by the 
imaging apparatus 2. In this process, the image processing 
apparatus 10 controls a character, a virtual object, to be 
superimposed on the game card 4 when displayed. The player 

25 can easily recognize whether the game card 4 is located 

within the imaging area 5 by watching the display 7. If the 
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game card 4 is not located within the imaging area 5, the 
player may allow the imaging apparatus 2 to capture the image 
of the game card 4 by shifting the position of the game card 
4 or by adjusting the orientation of the imaging apparatus 2. 
5 [0027] In the game application according to this example, 

the player moves a character by manipulating the game card 4. 
As called for by the nature of the game application, it is 
favorable that the player feels the sense of unity between 
the game card 4 and the character. For this purpose, the 
10 image of the character is superimposed on the game card 4. 
As the player moves the game card 4 slowly in the imaging 
area 5, the character tracks the movement of the game card 4, 
remaining placed on the game card 4 . 

[0028] The character motion as described above is 

15 controlled by the image processing apparatus 10. First, the 
image analysis apparatus 20 extracts image information for 
the game card 4 from the frame image captured by the imaging 
apparatus 2. The image analysis apparatus 20 further 
extracts the two-dimensional code printed on the game card 4 
20 from the image information for the game card 4. In this 

process, the image analysis apparatus 20 determines position 
information, orientation information and distance information 
for the game card 4 in a space, by referring to the two- 
dimensional code on the game card 4. 
25 [0029] The image processing apparatus 20 determines the 

distance between the imaging apparatus 2 and the game card 4, 
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the orientation of the game card 4 by using a reference cell 
and corner cells in the image of the game card 4 included in 
the fame image. The character is controlled so that it faces 
forward when displayed on the game card 4. The image 
5 analysis apparatus 20 also acquires code data embodied on the 
game card 4 by referring to an array of the multiple 
rectangular cells in a two-dimensional code. 

[0030] While Fig. 1 shows a state in which the game card 

4 is placed on a table 3, the game card 4 may be at an angle 

10 with respect to the table 3. Alternatively, the game card 4 
may be elevated from the table 3. The image analysis 
apparatus 20 has the function of re.cognizing the angled 
position of the game card 4 or variation in the height of the 
game card 4 with respect to the table 3, through image 

15 analysis. The result of image analysis by the image analysis 
apparatus 20 is sent to the game apparatus 30. The frame 
image captured by the imaging apparatus 2 may be sent to the 
game apparatus 30 for image analysis by the game apparatus 30. 
In this case, the image processing apparatus 10 may be formed 

20 only of the game apparatus 30. 

[0031] The game apparatus 30 controls the character to 

be displayed on the game card 4, by referring to the result 
of image analysis by the image analysis apparatus 20. The 
character may be associated with the code data extracted from 

25 the game card 4 and assigned to each game scene. In this 

case, when a game scene changes, the character displayed is 
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also changed. 

[0032] Fig. 2 shows a two-dimensional code printed on 

the surface of the game card 4. Multiple polygonal cells 
having predetermined shapes are arranged two-dimensionally 
5 according to a predetermined rule of arrangement. More 

specifically, the two-dimensional code embodied on the game 
card 4 is provided with a reference cell 100 that serves as a 
reference in recognizing the game card 4 in the image data 
captured by the imaging apparatus 2, a code data field 102 

10 constituting code data by containing a two-dimensional array 
of multiple rectangular cells 106, and multiple corner cells 
104a, 104b, 104c and 104d (hereinafter, generically referred 
to as corner cells 104) provided to surround the code data 
field 102. The code data field 102 and the corner cells 104 

15 are located in a code field 110 in the two-dimensional code. 
[0033] The reference cell 100, the rectangular cell 106 

and the corner cells 104 have predetermined shapes, 
respectively. In the two-dimensional code shown in Fig. 2, 
the cells are shown painted over with black. Referring to 

20 Fig. 2, an area 108 is formed by connecting multiple 

rectangular cells 106 and constitutes the code data in 
combination with the other rectangular cells 106. Each 
rectangular cell 106 has the identical shape (in this case, a 
square shape) . If several types of the game cards 4 

25 according to this example are available, the reference cell 
100 and the corner cells 104 are identically located and 
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sized in the different types. The game card 4 can be 
uniquely identified by referring to the arrangement of the 
rectangular cells 106 in the code data field 102. 
[0034] A description will now be given of the details of 

5 the two-dimensional code according to this example. Letting 
one side of the rectangular cell 106 of a square shape 
represent one block, the reference cell 100 is formed as a 
rectangular cell (element) with the longer side spanning 7 
blocks horizontally and the shorter side spanning 1.5 blocks 

10 vertically. A square area formed by a 1 by 1 block according 
to this example will be referred to as a block area. 
[0035] The code data field 102 is formed within an area 

defined by a 7 by 7 block. One side of the area is parallel 
with the longer side of the reference cell 100 and is removed 

15 from the longer side of the reference cell 100 by 1 block. 

As shown, the code field 110 is an area defined by an 8 by 8 
block. The code data field 102 is included in an area in 
which the code field 110 is formed. The rectangular cells 
106 are actually provided outside the 2 by 2 blocks at the 

20 four corners of the area defined by the 7 by 7 block. More 
specifically, the rectangular cells 106 are not provided in 
the 2 by 2 blocks at the four corners. Therefore, the 
rectangular cells 106 are provided in 33 (=7x7-2x2x4) block 
areas so as to form code data. 

25 [0036] A corner cell 104a is provided in an area at the 

top left corner of the code area 110. The area at the top 
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left corner may be within an area of 2 by 2 block at the top 
left corner of the area defined by the 7 by 7 block 
representing the code data field 102. Alternatively, the top 
left corner area may protrude from the area of 2 by 2 block. 
5 In the illustrated example, the corner cell 104a is provided 
so as to protrude from the area of 2 by 2 block at the top 
left corner in the code data field 102, by 0.5 blocks 
vertically and by 0.5 blocks horizontally. That is, the 
corner cell 104a is located within an area defined by a 2.5 

10 by 2.5 block at the top left corner of the code field 110. 

Similarly, the corner cell 104b is provided in an area at the 
top right corner of the code field 110. The corner cell 104c 
is provided in an area at the bottom right corner of the code 
field 110. The corner cell 104d is provided in an area at 

15 the bottom left corner of the code field 110. In the 

illustrated example, the corner cells 104 are formed as 
rectangles. More specifically, they are formed as squares 
with each side being equal to 1.5 blocks in length. 
[0037] Given that a block area represents 1 bit, the 

20 code data field 102 is capable of coding information 

comprising a total of 33 bits. Of the 33 bits, 9 bits 
represent check data for verifying whether the code data is 
proper. Therefore, information comprising 24 bits is coded 
in the code data field 102. 

25 [0038] The rectangular cells 106 are important elements 

that represent code data and should be accurately recognized 

SC05022WO0 (SC-70086WO) 



19 

by the image analysis apparatus 20. For this purpose, it is 
ideal that all the rectangular cells 106 be formed to be 
large. Naturally, however, due to the size constraints of 
the game card 4, enlarging the rectangular cells 106 would 
5 result in reduction in the number of rectangular cells 106 in 
the code data field 102. This is not favorable since it is 
equivalent to reducing the number of information bits. 
[0039] In this background, the corner cells 104 in this 

example are formed to occupy larger areas than the 

10 rectangular cells 106. In the image analysis apparatus 20 

according to this example, the four corner cells 104 are used 
to detect the code data field 102, as described later. This 
translates to the fact that, if the four corner cells 104 
fail to be detected, the rectangular cells 106 cannot be 

15 detected. Therefore, it is imperative in this two- 
dimensional code that the corner cells 104 be recognized 
properly prior to the recognition of the rectangular cells 
106. By forming the four corner cells 104 to be larger than 
the rectangular cells 106, the rate of recognition of the 

20 four corner cells 104 is increased. 

[0040] The image analysis apparatus 20 extracts 

candidates for the reference cell 100 in the frame image and 
assumes the one that meets a predetermined criteria as being 
the reference cell 100. The image analysis apparatus 20 then 

25 determines whether there are four corner cells 104 in the 

neighborhood of the cell assumed to be the reference cell 100. 
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If the four corner cells 104 are recognized, the image 
analysis apparatus 20 reads the layout of rectangular cells 
106 in an area surrounded by the four corner cells 104 (i.e. 
in the code data field 102). In this way, the image analysis 
5 apparatus 20 acquires code data. This confirms a 

determination that the cell assumed to be the reference cell 
100 is really the reference cell 100. The game card 4 is 
recognized as such. The image analysis apparatus determines 
by calculation the position and orientation of the game card 

10 4 in the virtual three-dimensional coordinate system, with 
reference to the imaging apparatus 2 as the origin. 
[0041] Fig. 3 shows the structure of the image analysis 

apparatus. The image analysis apparatus 20 is provided with 
a frame image acquirer 40, a real object extractor 42 and a 

15 state determiner 44. The real object extractor 42 is 

provided with a binarization processor 50, a reference cell 
detector 52, a code field detector 54 and a verification 
processor 56. Processing functions of the imaging apparatus 
20 according to this example are implemented by a CPU, a 

20 memory, a program loaded into the memory, etc. Fig. 3 

depicts function blocks implemented by the cooperation of the 
elements. The program may be built in the imaging apparatus 
20 or supplied from an external source in the form of a 
recording medium. Therefore, it will be obvious to those 

25 skilled in the art that the functional blocks may be 

implemented by a variety of manners including hardware only, 
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software only or a combination of both. In the illustrated 
example, the CPU of the image analysis apparatus 20 has the 
functions of the frame image acquirer 40, the real object 
extractor 42 and the state determiner 44. 
5 [0042] The frame image acquirer 4 0 acquires a frame 

image of a real space captured by the imaging apparatus 2. 
The imaging apparatus 2 captures a frame image at regular 
intervals. Preferably, the imaging apparatus 2 generates 
frame images at intervals of 1/60 seconds. 

10 [0043] The real object extractor 42 extracts a real 

object image, i.e., an image of the game card 4, from the 
frame image. More specifically, the binarization processor 
50 translates the frame image into a binary representation so 
as to represent the information for the frame image in ons 

15 and offs. The reference cell detector 52 extracts the 

reference cell 100 by referring to the binary data for the 
frame image. When the reference cell 100 is detected, the 
code field detector 54 detects the code' field 110 based upon 
the position of the reference cell 100. The verification 

20 processor 56 verifies the code data included in the code 
field 110. 

[0044] The state determiner 44 determines the state of 

the real object in the coordinate system. More specifically, 
the state determiner 44 determines the distance of the game 
25 card 4 from the imaging apparatus 2 as well as the position 

and orientation of the game card 4. The position information, 
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orientation information and distance information determined 
in the state determiner 44 are associated with the code data 
acquired from the two-dimensional code before being 
transmitted to the game apparatus 30. If multiple game cards 
5 4 are located within the imaging area 5, the position 

information, orientation information, distance information 
and code data are associated with each of the game cards 4 
before being transmitted to the game apparatus 30. Since the 
frame image captured by the imaging apparatus 2 is displayed 

10 on the display 7 of the game system 1, the frame image itself 
is transmitted to the game apparatus 30. The game apparatus 
30 reads a character associated with the code data and 
superimposes the character on the game card 4 displayed in a 
three-dimensional virtual space. 

15 [0045] Fig. 4 is a flowchart showing a processing 

procedure for recognition of two-dimensional codes. An image 
of a two-dimensional code is captured by the imaging 
apparatus 2. Resultant image data for one frame is acquired 
by the frame image acquirer 40. The binarization processor 

20 50 binarizes the image data (S10). In binarization, pixels 
having a luminance value higher than a predetermined 
threshold are coded into pixel values of "0" and represented 
in white when displayed. Hereinafter, pixels coded into 
pixel values of "0" will be referred to as white pixels. 

25 Pixels having a luminance value equal to or lower than the 

threshold are coded into pixel values of "1" and represented 
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in black when displayed. Hereinafter, pixels coded into 
pixel values of "1" will be referred to as black pixels. The 
binarization processor 50 organizes a set of contiguous black 
pixels into a linked area. The binarization processor 
5 numerically numbers (labels) multiple linked areas of black 
pixels thus extracted in the ascending order, starting with 1 
(S12) . For example, the linked areas may be numbered in the 
order of extraction. The numbers are stored in a RAM (not 
shown) along with the coordinate data. Since the corner 

10 cells 104 are larger than the rectangular cells 106 when 

displayed, the use of the two-dimensional code shown in Fig. 
2 will increase the likelihood of the corner cells 104 being 
detected as linked areas of black pixels and improve the 
accuracy of coordinate position. 

15 [0046] The reference cell detector 52 determines whether 

the display position of the two-dimensional code recognized 
in the previous process for recognizing two-dimensional codes 
(e.g., the position of the center point of the reference cell 
100) , which is targeted at image data different from the one 

20 currently being processed, is stored in the RAM (S14). If 
the center point of the reference cell 100 in the previous 
frame is stored in the RAM (Y in S14), the position stored is 
configured as a starting point for the reference cell 
detecting process (S16) . If the reference cell 100 was 

25 detected in the previous image frame, it is likely that the 
position of the game card 4 has changed little. Therefore, 
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by using the center point of the reference cell 100 in the 
previous frame, the reference 100 in the current frame may be 
detected efficiently. If the position of the two-dimensional 
code is not stored in the RAM (N in S14), the reference cell 
5 detector 52 configures the point at the center of the display 
7 as a starting point for the reference cell detection 
process (S18) . Following the establishment of the starting 
point, the reference cell detector 52 performs the process of 
detecting the reference cell 100 (S20) . 

10 [0047] Fig. 5 is a flowchart showing the process of 

detecting a reference cell. The pixel counts entered in the 
flowchart depend on, for example, the resolution of the 
imaging apparatus 2 . The pixel counts entered here are by 
way of example and are provided for ease of understanding. 

15 The reference cell detector 52 reads the total number M of 
linked areas of black pixels labeled in S12 of Fig. 4 and 
initializes the counter value j to 1 (S100) . Subsequently, 
the reference cell detector 52 starts searching for linked 
areas of black pixels at the starting point established in 

20 S16 or S18 of Fig. 4 and continues the search, in a 

counterclockwise spiral on the screen of the display 7. The 
first linked area of black pixels detected is selected as a 
reference cell candidate area (S102). The reference cell 
detector 52 extracts the edges of the reference cell 

25 candidate area thus selected and identifies the shorter side 
and the longer side (S104) . 
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[0048] The reference cell detector 52 determines whether 

the shorter side comprises a predetermined number of pixels 
(e.g., 20 pixels or more) (S106) . The shorter side of the 
reference cell 100 is designed to be 1.5 blocks long. Thus, 
5 if the shorter side of a reference cell 100 comprises less 
than 20 pixels, the length of 1 block will comprise fewer 
pixels. That will mean that the number of pixels in one side 
of the rectangular cell 106, the smallest constituting 
element in the two-dimensional code, will be 13 or fewer. In 

10 this case, the imaging apparatus 2 cannot properly captures 
an image. Therefore, if the shorter side is determined to 
comprise less than 20 pixels (N in S106) , it is determined 
that the linked area of black pixels selected in S102 is not 
the reference cell 100, whereupon control proceeds to S116. 

15 [0049] If the shorter side is determined to comprise 20 

pixels or more (Y in S106) , the reference cell detector 52 
determines whether the longer side of the reference cell 
candidate area comprises a predetermined number of pixels 
(e.g., 300 pixels or fewer) (S108). Given that the longer 

20 side of a reference cell 100 comprises, for example, more 

than 300 pixels, the length of 1 block, determined as 1/7 of 
the length of the longer side, will grow so large that the 
bottom right cell 104c and the bottom left cell 104d 8.5 
blocks away from the reference cell 100 will fail to be 

25 captured by the imaging apparatus 2. Therefore, if the 

longer side is determined to comprise more than 300 pixels (N 
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in S108), it is determined that the linked area of black 
pixels selected in S102 is not the reference cell 100, 
whereupon control proceeds to S116. 

[0050] If it is determined that the longer side 

5 comprises 300 pixels or less (Y in S108) , the reference cell 
detector 52 determines whether the total number of black 
pixels in the reference cell candidate area is no less than 
20 and less than 1500 (S110) . If the total number of black 
pixels is less than 20 (N in S110), the same problem as 

10 described above, occurring if the shorter side comprises less 
than 20 pixels, arises. If the total number of black pixels 
is 1500 or more (N in S110), the same problem as described 
above, occurring if the longer side comprises 300 pixels or 
more, arises. Accordingly, a determination is made in these 

15 cases that there is little likelihood that the reference cell 
candidate area is the reference cell 100, whereupon control 
proceeds to S116. 

[0051] If the total number of black pixels in the 

reference cell candidate area is no less than 20 and less 

20 than 1500 (Y n S110), the reference cell detector 52 

determines the rectangularity of the reference cell candidate 
area (S112). The rectangularity is determined by referring 
to the moment of the reference cell candidate area, as 
described in patent document 1. If it is determined that the 

25 area is not rectangular (N in S112), it is determined that 
the reference cell candidate area is not the reference cell 
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100, whereupon control proceeds to S116. If it is determined 
that the reference cell candidate area is rectangular (Y in 
S112), the reference cell detector 52 establishes the 
reference cell candidate area as the reference cell 100 
5 (S114) and stores in the RAM the label number assigned to the 
linked area of black pixels determined as the reference cell 
candidate area. In this way, the reference cell 100 is 
detected. If the image data includes only one game card 4, 
the process for detection of the reference cell is terminated 
10 when the reference cell 100 is detected. If there are 

multiple game cards 4 are possibly included, control proceeds 
to S116- 

[0052] The reference cell detector 52 determines whether 

a count j is equal to the total number M of the reference 
15 cell candidate areas (S116). If the " count j has not reached 
M (N in S116) , the count j is incremented by 1 (S118) , 
whereupon control proceeds to S102. If the count j has 
reached M (Y in S116) , the reference cell detection process 
is terminated. 

20 [0053] Referring back to Fig. 4, if the reference cell 

100 is not detected in the reference cell detection process 
of S20 (N in S22), it means that the game card 4 is not 
identified, whereupon the process for recognition of the two- 
dimensional code is terminated. If the reference cell 100 is 

25 detected (Y in S22), the process of detecting a code field is 
performed (S24) . 
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[0054] Fig. 6 is a flowchart showing the process of 

detecting a code field. The code field detector 54 reads the 
total number N of reference cells 100 established in S114 of 
Fig. 5 and initializes a count k to 1 (S200) . It will be 
5 assumed here that the reference cells 100 established in S114 
are numbered in the order of establishment. Subsequently , 
the code field detector 54 reads the total number M of the 
linked areas of black pixels labeled in S12 of Fig. 4 and 
initializes the count j to 1 (S202) . The code field detector 
10 54 detects the linked area of black pixels corresponding to 
the count j and selects the area as a bottom left cell 
candidate area (S204). 

[0055] The code field detector 54 then determines 

whether the bottom left cell candidate area thus selected is 

15 located within a search range predefined with respect to the 
reference cell 100 detected in S20 of Fig. 4 and assigned a 
number corresponding to the count k (S206) . The positional 
relationship between the reference cell 100 and the bottom 
left cell 104d is predefined as shown in Fig. 2. Accordingly, 

20 it is possible to narrow a range of search for the bottom 
left cell by referring to the coordinate data for the 
reference cell 100. Since the reference cell 100 is a 
horizontally elongated rectangle, the search range may be 
limited to two positions opposite to the longer side. If the 

25 corner cell candidate area is not located within the search 
range (N in S206), it is determined that the associated 
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linked area of black pixels is not the bottom left cell 104d, 
whereupon control proceeds to S232. If the corner cell 
candidate area is located within the search range (Y in S206) , 
the associated linked area of black pixels is established as 
5 . the bottom left cell 104d (S208) . 

[0056] Following the establishment of the bottom left 

corner cell 104d, the code field detector 54 initializes to 1 
a count 1, another counter value indicating the number 
identifying a linked area of black pixels (S210) . The code 
10 field detector 54 detects a linked area of black pixels 

assigned the number corresponding to the count 1. The code 
field detector 54 selects the area as a bottom right cell 
candidate area (S212). 

[0057] Subsequently, the code field detector 54 

15 calculates a ratio between the number of pixels (occupied 
area) in the bottom left cell 104d established in S208 and 
the number of pixels (occupied area) in the bottom right 
candidate area selected in S212. The code field detector 54 
determines whether the ratio (area ratio) is, for example, no 
20 less than 1/6 and no more than 6 (S214). The number of 

pixels in the corer cells 104 acquired in imaging varies with 
the distance from the imaging apparatus 2. More specifically, 
the closer to the imaging apparatus 2, the larger the area. 
The more distant, the smaller the area. For this reason, 
25 areas represented in the image data may differ from corner 

cell to corner cell. However, it is normally quite unlikely 

SC05022WO0 (SC-70086WO) 



30 

that the ratio between the bottom left cell and the bottom 
right cell is below 1/6 or exceeds 6 (N in S214) . A 
candidate that yields such a ratio cannot be identified as a 
corner cell in a given two-dimensional code and prompts the 
5 control to proceed to S226. 

[0058] If the area ratio is no less than 1/6 and no more 

than 6 (Y in S214), the code field detector 54 determines 
whether the distance between the center point in the bottom 
left cell 104d and the center point in the bottom right cell 

10 candidate area selected in S212 fulfills a predefined 

condition (S216) . The predefined condition may be that the 
distance between the center points approximates the length of 
the longer side of the reference cell 100. If the predefined 
condition is not fulfilled (N in S216) , control proceeds to 

15 S226. If the predefined condition is fulfilled (Y in S216), 
the code field detector 54 establishes the linked area of 
black pixels as the bottom right cell 104c (S218) 
[0059] Once the left bottom cell 104d and the right 

bottom cell 104c are established, the code field detector 54 

20 subjects the reference cell 100, the bottom left cell 104d, 

the right bottom cell 104c to affine transformation in the X- 
direction and' Y-direction on the screen of the display 7 
(S220) . The length of 1 block is calculated based upon the 
length of the longer side or the shorter side of the 

25 reference cell 100 established in S20. The code detector 54 
generates a code map from the image obtained by affine 
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transformation, by mapping, into cells, the linked areas of 
black pixels included in the code field 110 (S222) . 
[0060] The code detector 54 identifies the four corner 

cells 104 from among the cells in the code map thus generated. 
5 The code detector 54 then determines whether 3 block areas 

surrounding the corer cells 104 comprise white pixels (S224) . 
If the 3 block areas surrounding the corner cells 104 are 
comprised of white pixels (Y in S224), the code map is 
established as the code field 110 in the two-dimensional code 
10 (S230) , whereupon control proceeds to S236. If the 3 block 
areas surrounding the corner cells 104 are not comprised of 
white pixels, control proceeds to S226. 

[0061] In S226, a determination is made as to whether 

the count i is equal to the total number M of the linked 
15 areas of black pixels. If the count i is equal to M (Y in 
S226), control proceeds to S232. If not (N in S226) , the 
count i is incremented by 1 (S228), whereupon control is 
returned to S212. 

[0062] In S232, a determination is made as to whether 

20 the count j is equal to the total number M of the linked 

areas of black pixels. If the count j is equal to M (Y in 
S232), control proceeds to S236. If not (N in S232), the 
count j is incremented by 1 (S234), whereupon control is 
returned to S204. 
25 [0063] In S236, a determination is made as to whether 

the count k is equal to the total number N of the reference 
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cells 100. If the count k is not equal to N (N in S236) , the 
count k is incremented by 1 (S238), whereupon control is 
returned to S202. In this way, multiple two-dimensional 
codes in the image data can be identified through the search. 
5 If the count k is equal to N (Y in S236) , the process for 
detection in the code field 110 is completed. 

[0064] Referring to Fig. 4, if the code field 110 is not 

detected in the code field detection process of S24 (N in 
S26) , it means that no two-dimensional codes are identif ied, 

10 whereupon the process for recognition of two-dimensional 

codes is terminated. If the code field 110 is detected (Y in 
S26) , the process of verifying code data is performed (S28). 
[0065] Fig. 7 is a flowchart showing the process of 

verifying code data. As already mentioned, the 33-bit code 

15 data comprises 9-bit check data which is generated according 
to a predetermined algorithm and 24-bit information data. In 
the verification process, the code data is verified using the 
check data. The process of verifying code data proceeds by 
generating check data from the code data according to the 

20 algorithm and checking the check data thus generated against 
the check data in the code data. 

[0066] Initially, the verification processor 56 

initializes to 1 a count p, which indicates the number of 
times that reference values calculated in S304 and S308 
25 (described later) are shifted 1 bit to the right (S300) . 

Subsequently, the verification processor 56 calculates the 
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values of the code data and the check data by referring to 
the code map in the code field 110 (S302) . 
[0067] The verification processor 56 obtains an 

excusive-OR of the code data value thus calculated (bit 
5 stream) and OxFFFFFF (S304) . The resultant value (bit 

stream) is designated as a reference value (reference bit 
stream). A determination is made as to whether the least 
significant bit (LSB) of the reference bit stream is set to 
"1" (S306) . If the LSB is not set to "1" (N in S306) , the 

10 verification processor 56 obtains an exclusive-OR of the 
reference value (reference bit stream) and 0x8408 (S308) . 
The resultant value (bit stream) is designated as a new 
reference value (reference bit stream) , whereupon control 
proceeds to S310. When the LSB is set to "1" (Y in S306) , 

15 control similarly proceeds to S310. 

[0068] The verification processor 56 shifts the 

reference values (reference bit stream) calculated in S304 or 
in S308 1 bit to the right (S310) and determines whether the 
count p' is equal to a predetermined number of shifts, which 

20 is 24 (S312). If it is determined that p^24 (N in S312), the 
count p is incremented by 1 (S314), whereupon control 
proceeds to S306. 

[0069] If it is determined that p=24 (Y in S312), the 

calculated bit stream is ANDed with OxlFF (S316) . The 
25 verification processor 56 determines whether the result 

obtained by the logical AND operation is equal to the check 

SC05022WO0 (SC-70086WO) 



34 

data value calculated (S318) . If it is determined that the 
result from the logical AND operation is equal to the check 
data value (Y in S318) , the verification processor 56 
determines that the code field 110 detected in S24 of Fig. 4 
5 has a proper two-dimensional code pattern, verifying the code 
field 110 of the two-dimensional code (S320) . If the result 
of the logical AND operation does not match the check code 
value (N in S318), the verification processor 56 determines 
that the reading of the code field 110 is in error, whereupon 
10 the process of verifying code data is terminated. While Fig. 
7 only shows the process of verifying one code field 110, the 
verification process is performed for each of the multiple 
code fields 110 if detected. 

[0070] Referring back to Fig. 4, if the code field 110 

15 is not verified (N in S30) in the process of detecting code 
data of S28, it means that no two-dimensional codes are 
identified, whereupon the process for recognition of two- 
dimensional codes is terminated. If the code field 110 is 
verified (Y in S30), the code data value (i.e. the value of 
20 the two-dimensional code) is stored in, for example, the RAM 
and maintained therein (S32) . This completes the process for 
recognition of two-dimensional codes. 

[0071] A description will now be given of how the 

position and orientation of the game card 4 in the three- 
25 dimensional coordinate system are determined. To obtain the 
position and orientation of the game card 4 in the virtual 
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three-dimensional coordinate system, the angle of view of the 
imaging apparatus 2, the screen resolution, the coordinate 
position of the four corner cells and the actual distance 
between the corner cells 104 are used in this example. The 
5 term "screen" refers to the screen of the display 7 

displaying the image captured by the imaging apparatus 2. 

[0072] 

(Step 1) 

Initially, the distance between a viewpoint and a plane 
10 of projection (screen) is determined. The distance between 
the viewpoint and the plane of projection (screen) is 
determined by referring to the angle of view of the camera 
and the screen resolution. 

Assuming that the horizontal angle of view of the camera is 
15 denoted by 9, the horizontal screen resolution is denoted by 

W and the distance between the viewpoint and the plane of 

projection (screen) is denoted by P, the following 

relationship holds: 

P=(W*0.5) /tan(9*0.5) 
20 [0073] 

(Step 2) 

A three-dimensional vector pointing from the viewpoint 
to each of the corner cells 104 will be determined next. 

Given that the screen coordinate position of a corner 
25 cell is denoted by SX and'SY, the distance between the 

viewpoint and the plane of projection (screen) is denoted by 
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P, and the three-dimensional vector V pointing to the corner 
cell is denoted by V, the following relationship holds. 
V=(SX, SY, P) 

The screen coordinate position is defined with respect 
5 to the center of the screen as the origin. 
[0074] 
(Step 3) 

Normal vectors normal to a plane formed by three points 
including the viewpoint and two adjacent corner cells are 

10 determined. A total of four normal vectors are generated. 
Given that a three-dimensional vector pointing from the 
viewpoint to the corner cell 104a is denoted by VI, a three- 
dimensional vector pointing from the viewpoint to the corner 
cell 104b is denoted by V2, a three-dimensional vector 

15 pointing from the viewpoint to the corner cell 104c is 

denoted by V3, a three-dimensional vector pointing from the 
viewpoint to the corner cell 104d is denoted by V4, a normal 
vector normal to a plane formed by the viewpoint and the 
corner cells 104a and 104b is denoted by S12, a normal vector 

20 normal to a plane formed by the viewpoint and the corner 

cells 104b and 104c is denoted by S23, a normal vector normal 
to a plane formed by the viewpoint and the corner cells 104c 
and 104d is denoted by S34, and a normal vector normal to a 
plane formed by the viewpoint and the corner cells 104d and 

25 104a is denoted by S41, the following relationship holds. x 
S12=VlxV2 
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S23=V2xV3 
S34=V3xV4 
S41=V4xVl 

where "x" denotes a cross product. 
5 [0075] 

(Step 4) 

Subsequently , the orientation (coordinate axes) of the 
game card 4 in the three-dimensional space is determined. 
The coordinate axes of the card are determined based upon the 

10 four normal vectors determined in step 3. 

Given that the X axis of the local coordinate system of 
the card (running in the horizontal direction on the surface 
of the card) is denoted by VX, the Y axis of the local 
coordinate system of the card (running in the direction 

15 passing through the surface of the card) is denoted by VX, 
and the Z axis of the local coordinate system of the card 
(running in the vertical direction on the surface of the 
card) is denoted by VZ, the following relationship holds. 
VX=S12xS34 

20 VZ=S23xS41 
VY=VZxVX 
[0076] 
(Step 5) 

Finally, the position of the game card 4 in the three- 
25 dimensional space is determined. In step 4, a matrix for 
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converting into local coordinates of the game card 4 is 
obtained. The position of the card is easily determined if 
the actual distances between the corner cells are known. 
Using the position of the corner cell 104a in the three- 
5 dimensional coordinate system as a variable, the positions of 
the corner cells 104b, 104c and 104d in the three-dimensional 
coordinate system can be represented using the variable. 
Since the positions of the four corner cells in the screen 
coordinate system are known, the position of the game card 4 
10 in the three-dimensional coordinate system is identified by 

finding a solution to simultaneous equations of the resultant 
expressions . 

[0077] As described above, the use of the two- 

dimensional code shown in Fig. 2 helps improve the rate of 

15 recognition of the code field 110; Moreover, the positions 
of the corner cells 104 in the two-dimensional coordinate 
system can be accurately determined. More specifically, by 
forming the corner cells 104 to be relatively large, the 
center positions of the corner cells 104 are determined 

20 accurately so that the position of the game card 4 is 

determined accurately. By determining the center positions 
of the multiple corner cells 104 accurately, the positional 
relationship between the reference cell 100 and the corner 
cell 104 is known accurately so that the precision in affine 

25 transformation is improved. Thereby, the position and 

orientation of the game card 4 can be determined accurately. 
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[0078] Fig. 8 shows another example of two-dimensional 

code printed on the surface of a game card. The reference 
cell 100 having a predetermined shape and the code field 110 
are provided on the surface of the game card 4 . Compared 
5 with the two-dimensional code of Fig. 2, the two-dimensional 
code shown in Fig. 8 is designed such that the multiple 
corner cells 104a, 104b, 104c and 104d surrounding the code 
data field 102 are circular in shape. As mentioned already, 
the two-dimensional coordinate position of a corner cell 104 

10 is identified by the center point of the corner cell 104. By 
configuring the corner cell 104 to have a circular shape, it 
is ensured that the shape of the corner cell 104 imaged by 
the imaging apparatus 2 remains unchanged irrespective of the 
orientation of the game card 4 with respect to the imaging 

15 apparatus 2. Consequently, the center point of the corner 
cell 104 does not vary with the orientation with respect to 
the imaging apparatus 2. Therefore, the center point can be 
acquired consistently. This enables accurate determination 
of the position information, the orientation information and 

20 the distance information of the game card 4. The rectangular 
cells 106 may be formed to have a circular shape. Not only 
by configuring the corner cells 104 to have a circular shape 
but by forming the corner cells 104 to be larger than the 
rectangular cells 106, as described with reference to Fig. 2, 

25 the coordinate position of the corner cells 104 can be 
determined with higher precision. 
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[0079] Fig. 9 shows another example of two-dimensional 

code printed on the surface of a game card. The reference 
cell 100 and the code field 110 are provided on the surface 
of the game card 4. Compared with the two-dimensional code 
5 of Fig. 2, the two-dimensional code of Fig. 9 is designed 
such that at least one of the following cells is colored 
differently from the other cells: a) the reference cell 100 
having a predetermined shape, b) the multiple rectangular 
cells 106 constituting code data by forming a two-dimensional 

10 array, and c) the multiple corner cells 104 provided to 

surround the code data field 102. For example, red may be 
assigned to the reference cell 100, green to the rectangular 
cells 106 and blue to the corner cells 104. What is 
essential is that the color assigned can be imaged by the 

15 imaging apparatus 2 and binarized with reference to a 

predetermined threshold. The color need not be visible to 
the human eye. If the imaging property of the imaging 
apparatus 2 is known in advance, the cells may be colored in 
accordance with the property. 

20 [0080] A prerequisite for the process for recognition of 

two-dimensional codes according to this example is that the 
reference cell 100 is detected first as shown in S20 of Fig. 
4. The load incurred in processing the detection of the 
reference cell is high in the context of overall process for 

25 recognition of two-dimensional codes. Accordingly, it is 
preferable that the reference cell 100 be efficiently 
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detected in the image data. Of the multiple types of cells, 
it is thus preferable that the at least the reference cell 
100 be assigned a color different from the rectangular cells 
106 and the corner cells 104. 
5 [0081] An RGB color filter is provided in front of the 

light receiving surface of the imaging apparatus 2. Each 
frame image is represented by pixel values in the RGB format 
scaled to a 256 grayscale. In this example, the binarization 
processor 50 has the function of setting up a range of pixel 

10 values in the RGB format for the purpose of binarization. 

Given that the reference cell 100 is painted over with red, 
the binarization processor 50 sets up a range of pixel values 
in the RGB format for binarization of the reference cell, in 
the binarization process in S10 of Fig. 4. For example, the 

15 binarization processor 50 sets up a range 200-255 of red (R) 
pixel values, a range 0-100 of green (G) pixel values and a 
range 0-100 of blue (B) pixel values. The binarization 
processor 50 extracts pixels in the composite RGB format 
within these ranges and codes their pixel values as binary 

20 "Is". For example, a pixel in the composite format having an 
R pixel value of 225, a G pixel value of 30 and a B pixel 
value of 50 is coded as binary "1" and assigned the color 
black in a binary image. If a pixel value in the composite 
RGB format does not fit within these ranges, the pixel value 

25 is coded as binary "0" and the pixel is assigned the color 
white in a binary image. Extracting red pixels from the 
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image data as described above enables linked areas 
represented as black pixels in a binary image to be labeled 
in S12 of Fig. 4. By targeting a particular color (in this 
case, red) for extraction of selected pixels in image data as 
5 black pixels, the number of noises extracted is reduced. 
Time required for the reference cell detector 52 to detect 
the reference cell 100 in the binary data for the frame image 
is also reduced. 

[0082] Since the pixel values are affected by the 

10 property of a color filter, the red pixel values of the 

reference cell 100 may be examined while the game card 4 is 
being placed on the table 3, in order to establish pixel 
value ranges that allow for the property of a color filter. 
The above process enables determining the pixel values of the 
15 reference cell 100 in the RGB format acquired by the imaging 
apparatus 2. By subjecting the pixel values thus determined 
to binarization, tolerating a certain margin for error, the 
number of linked area of black pixels detected is 
significantly reduced. Since real-time processing is desired 
20 in the game system 1, efficient extraction of the reference 
cell 100 offers significant benefits. 

[0083] The aforementioned procedure can also deal with 

other types of cells (e.g., the corner cells 104 or the 
rectangular cells 106) which are colored. If the corner 
25 cells 104 are colored blue, the binarization processor 50 

establishes the RGB pixel value ranges for the corner cells 
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in S10 of Fig. 4. For example, the binarization processor 50 
may set up a range 0-100 of red (R) pixel values, a range 0- 
100 of green (G) pixel values and a range 200-255 of blue (B) 
pixel values. The binarization processor 50 extracts pixels 
5 in the composite RGB format within these ranges and codes 
their pixel values as binary "Is". For example, a pixel in 
the composite format having an R pixel value of 20, a G pixel 
value of 30 and a B pixel value of 240 is coded as binary "1" 
and assigned the color black in a binary image. If the pixel 

10 value in the composite RGB format does not fit within these 
ranges, the pixel value is coded as binary "0" and the pixel 
is assigned the color white in a binary image. Extracting 
blue pixels from the image data as described above enables 
linked areas represented as black pixels in a binary image to 

15 be labeled in S12 of Fig. 4. 

[0084] As described, an image predominantly bluish in 

color is binarized for conversion into a black image and 
labeled in S12 as a linked area of black pixels, before being 
selected by the code field detector 54 as a corner cell 

20 candidate area as shown in the flow of Fig. 6. By targeting 
a particular color (in this case, blue) for extraction of 
selected pixels in image data as black pixels, the number of 
noises extracted is reduced. Time required for the code 
field detector 54 to detect the corner cells 104 in the 

25 binary data for the frame image is reduced. If the 
rectangular cells 106 are colored green, an image 
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predominantly greenish in color is binarized for conversion 
into a black image in S10 of Fig. 4 and labeled as a linked 
area of black pixels in S12 before being used to generate a 
code map in S222 of Fig. 6. If the three types of cells are 
5 colored differently, the number of times that binarization 
processes are carried out is increased. Overall, however, 
the efficiency is improved since the processing volume 
required to search for the linked area after binarization is 
significantly reduced. 
10 [0085] The two-dimensional code shown in Fig. 9 is 

provided with colored cells. The colored cells may be formed 
to be large. Alternatively, they may be formed to be 
circular in shape, as shown in Fig. 8. 

[0086] Fig. 10 shows another example of two-dimensional 

15 code printed on the surface of a game card. The reference 
cell 100 and the code field 110 are provided on the surface 
of the game card 4. The two-dimensional code shown in Fig. 
10 is an embodiment of the two-dimensional code shown in Fig. 
9. The two-dimensional code shown in Fig. 10 is provided 
20 with the reference cell 100 having a predetermined shape, the 
multiple rectangular cells 106 constituting code data by 
forming a two-dimensional array, and the multiple corner 
cells 104 provided to surround the code data field 102. The 
four corner cells 104a, 104b, 104c and 104d are identically 
25 colored, but are colored " differently from the reference cell 
100 and the rectangular cells 106. 
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[0087] For improvement of the appearance of the game 

card 4, the corner cells 104 are aligned to the respective 
side edges of the game card 4. The corner cells 104 are 
provided in proximity to the four corners of the game card 4 
5 and are formed to be larger than the rectangular cells 106. 
As such, the four corner cells 104 stand out from the other 
features on the game card 4. The corner cells 104 are 
originally intended as provisions to specify the location of 
the rectangular cells 106 but the appearance thereof should 

10 also count so long as the game card 4 is used as such. In 

this respect, the border edge 107 delineating the game card 4 
is designed with a twist such that it curves inward as it 
reaches the corner cells 104 and skirts the corner cells 104, 
resulting in the sophisticated design of the corner cells 104. 

15 [0088] When the corner cells 104 are aligned to the 

respective side edges of the game card 4, it is preferable 
that the corner cells 104 are assigned a color other than 
black. Assuming that the corner cells 104 are colored black, 
imaging of the game card 4 by the imaging apparatus 2 and 

20 subsequent binarization may result in shadows, occurring at 
the edge of the game card 4, being converted into black 
pixels, with the result that the two corner cells 104 
converted into black pixels being linked to each other via 
the shadows at the edge of the game card 4. If the two corner 

25 cells 104 are linked through binarization, it will be 

difficult to isolate the two corner cells 104 from the linked 
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black pixels. This calls for assigning a color other black 
to the corner cells 104 and setting up the range of RGB 
threshold values (i.e., the ranges of RGB values) for 
binarization appropriately, in order to avoid a situation in 
5 which corner cells 104 are linked due to a shadow at the edge 
of the game card 4. As described, in case the corner cells 
104 are aligned to the respective side edges of the game card 
4, the corner cells 104 can be properly extracted by 
assigning thereto a color other than that of shadows, i.e., a 

10 color other than black. 

[0089] In the game card 4 shown in Fig. 10, the corner 

cells 104 are formed to be triangular in shape. The corner 
cells 104 may be formed as regular triangles. By forming the 
corner cells 104 as regular triangles, the precision in 

15 recognition of the corner cells 104 is improved. 

[0090] Figs. 11A-11C are charts showing a method of 

determining whether black pixels extracted represent a corner 
cell 104. As already described, a determination on the 
corner cells 104 is made by referring to a linked area of 

20 black pixels identified as a result of binarization of a 

frame image. A binarized frame image can be represented in a 
two-dimensional X-Y coordinate system. Therefore, by using 
coordinate values representing the linked area, a 
determination can be made as to whether the linked area 

25 represents the corner cell 104. 

[0091] Fig. 11A shows an example of linked area 
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extracted from a frame image. The width of a linked area 
109a in the X -axis direction is denoted by Hi, and the width 
in the Y-axis direction is denoted by Vi. The width is 
represented by the number of pixels in the two-dimensional 
5 coordinate system. Given that the number of pixels in the 
linked area 109a is denoted by d", whether the linked area 
109a is a candidate for the corner cell 104 is made by using 
the following expression. 
axHixVx^Ci^pxHixVi (a<p<l) 

10 For example, a is equal to 0.3 and p is equal to 0.7. 

If the number of pixels Ci in the linked area 109a 
fulfills the above expression, a determination may be made 
that the linked area 109a is a candidate for the corner cell 
104. Naturally, the constants a and (3 may be set to other 

15 values. 

[0092] Fig. 11B shows another example of linked area 

extracted from a frame image. Whether a linked area 109b is 
a candidate for the corner cell 104 is similarly determined 
according to the following expression. 

20 axH 2 xV 2 ^C 2 <pxH 2 xV 2 (a<P<l) 

where C 2 denotes the number of pixels in the linked area 109b, 
H 2 denotes the width of the linked area 109b in the X-axis 
direction, and V 2 denotes the width in the Y-axis direction. 
The linked areas shown in Figs. 11A and 11B are those 

25 that fulfill the discriminant and are determined to be 

candidates for the corner cell 104. As shown in Figs. 11A 
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and 11B, the angle formed by the game card 4 in the imaging 
area 5 as it is captured in a frame image may vary. Using 
the aforementioned discriminant enables the triangular corner 
cells 104 to be recognized properly. 
5 [0093] Fig. 11C shows another example of linked area 

extracted from a frame image. In this case, the number of 
pixels C 3 in the linked area 109c is smaller than axH 3 (width 
in the X-axis direction) xV 3 (width in the Y-axis direction). 
That is, the relationship C 3 <axH 3 xV 3 holds. Since this does 
10 not fulfill a discriminant (axH 3 xV 3 <C 3 <£xH 3 xV 3 ) , the linked 
area 109c shown in Fig. 11C is determined not to be the 
corner cell 104. 

[0094] As described, by forming the corner cells 104 as 

triangular areas, candidates for the corner cells 104 can be 

15 easily extracted, using the coordinate values in the two- 
dimensional coordinate system. By forming the corner cell 
104 as regular triangles, direction dependence of the 
determination process using the discriminant is reduced as 
compared to the other triangular shapes. Accordingly, the 

20 precision in recognizing the corner cells 104 is further 
improved. 

In the game card 4 shown in Fig. 10, the code data 
field 102 made up of the rectangular cells 106 are provided 
between the corner cells 104c and 104d toward the bottom of 
25 the card. This will create a space in the center of the game 
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card 4 where a picture of a character may be printed. 
[0095] Fig. 12 shows another example of two-dimensional 

code printed on the surface of a game card. Similar to the 
game card 4 shown in Fig. 10, the game card 4 shown in Fig. 
5 12 is designed so that the corner cells 104 are aligned to 
the respective edges of the game card 4 . The corner cells 
104 are formed to be triangular in shape and are assigned 
with a color different from that of the reference cell 100 
and code data fields 102a and 102b. 

10 [0096] In the game card 4 shown in Fig. 12, there are 

two mutually isolated areas in which are provided the 
rectangular cells 106. In this case, the two code data 
fields 102a and 102b are provided outside the area surrounded 
by the four corner cells 104a, 104b, 104c and 104d. The code 

15 field 102a is formed between the reference cell 100 and the 
top edge of the game card 4. The code field 102b is formed 
between a line connecting the corner cells 104c and 104d and 
the bottom edge of the game card 4. The line connecting the 
corner cells 104c and 104d is a line connecting the vertices 

20 of the triangles facing inward. By providing the code data 
field 102a and the code data field 102b outside the area 
surrounded by the four corner cells 104, a relatively large 
space is. made available in the center of the game card 4, 
where an image of a character may be printed. Each of the 

25 code data field 102a and the code data field 102b can contain 
12-bit information. Therefore, in total, 24-bit information 
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can be contained. It is preferable that the code data field 
102b provided toward the bottom of the game card 4 not 
include a predetermined sequence of colored bits or longer 
(e.g., a sequence of 3 bits or longer) in order to prevent it 
5 from being misidentif ied as the reference cell 100. 

[0097] When the rectangular cells 106 of Fig. 12 are 

targets of recognition, the reference cell 100 and the bottom 
left and bottom right corner cells 104d and 104c have already 
been detected. Therefore, the code data fields 102a and 102b 

10 can easily be detected by searching the neighborhood area. 
More specifically, the code data field 102a is located 
opposite to the corner cells 104c and 104d with respect to 
the reference cell 100. The code data field 102b is located 
opposite to the reference cell 100 with respect to the corner 

15 cells 104c and 104d. By providing the reference cell 100 and 
the code data field 102a in proximity to each other and by 
providing the corner cells 104c and 104d in proximity to the 
code data field 102b, efficiency of the search for the code 
data fields 102 is improved. 

20 [0098] While Fig. 12 shows the game card 4 in which the 

code data fields 102a and 102b are isolated, the code data 
fields 102a and 102b may be provided adjacent to each other. 
That is, the code data fields 102a and 102b may be provided 
between the reference cell 100 and the top edge of the game 

25 card 4. Alternatively, they may be provided between the 

corner cells 104d and 104c and the bottom edge of the game 
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card 4 . 

[0099] In the above description, it is assumed that the 

game card 4 is moved on the table 3. Alternatively, a game 
mat may be placed on the table 3 so that the game card 4 is 
5 moved on the mat . 

Fig. 13 shows a game mat on which to place a game card. 
Referring to Fig. 1, the game mat 300 is placed on the table 
3 so as to be located within the imaging area 5 of the 
imaging apparatus 2. The game mat 300 is fabricated of a 

10 soft material such as felt cloth. Preferably, the game mat 

can be rolled into a compact size for storage when not in use. 
[0100] A total of 9 cells 302a, 302b, 302c, 302d, 302e, 

302f, 302g, 302h and 302i (generically referred to as "cells 
302") are formed by multiple partition lines 304. The game 

15 mat 300 is formed to adapt to the game application for which 
it is designed. Each cell 302 marks an area in which the 
game card 4 is placed. When a user places the game card 4 in 
one of the cells 302, the image analysis apparatus 20 may 
identify the cell 302 in which the card is placed and may 

20 recognize the two-dimensional code on the game card 4 so that 
a game story that depends on the cell in which the card is 
placed may develop. In the game mat 300 shown in Fig. 13, a 
3x3 array of cells 302 is formed. Alternatively, other 
numbers of cells may be formed. Referring to the area in 

25 which the user can manipulate the game card 4 as a "play 

area", the square area in which the cells 302 are formed in 
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the game mat 300 shown in Fig. 13 represents a play area 306. 
It is not essential that the entirety of the game mat 300 be 
placed within the imaging area 5, whereas the play area 306 
should at least be located within the imaging area 5. In a 
5 game application in which the user can manipulate the game 
card 4 at will, the cells 302 need not be formed. 
[0101] Gradation areas 310a, 310b, 310c and 310d 

(hereinafter, generically referred to as gradation areas 310) 
characterized by gradual change in brightness are provided at 

10 the four corners of the player area 306 on the game mat 300. 
A gradation area 310 is formed as gradation of brightness 
levels in a given hue (one of the three color attributes--hue, 
brightness and saturation) . Saturation need not be varied. 
The gradation areas 310 are bounded by circles at their outer 

15 edge. Brightness levels vary gradually and concentrically 
from high to low from the center of the gradation area 310 
toward the outer edge thereof. 

[0102] Fig. 14 shows the appearance of the gradation 

area. The gradation area 310 is comprised of a single hue, 

20 in which the brightness varies gradually. Hue represents 

difference in shades such as red, green and blue. Brightness 
represents difference in brilliance. It will be assumed here 
that the gradation area 310 has a hue, i.e., is expressed in 
a chromatic color. Achromatic colors such as white, black 

25 and grey do not have a hue. The gradation area 310 may also 
be configured to have gradual variation in brightness from 
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white to black. The difference in the brightness in the 
gradation area 310 is acquired as difference in luminance 
when digitized by an RGB filter. 

[0103] Thus, the gradation area 310 is designed such 

5 that its brightness level is low at the outer edge 312 and 
high at the center 314. That is, the gradation area 310 is 
designed to have an area of intense chromatic color at the 
outer edge and an blank area toward the center. The four 
gradation areas 310 may be assigned the same color. By 

10 assigning a different color to one of the gradation areas 310, 
the difference may serve as an index for determining the 
orientation of the game mat 300. For example, the three 
gradation areas 310a, 310b and 310c may be assigned the same 
hue, and the other gradation area 310d may be assigned a 

15 different color. Upon successful detection of the four 

gradation areas 310, the orientation of the game mat 300 is 
identified since the position of the gradation area 310d with 
respect to the gradation areas 310a, 310b and 310c is known. 
[0104] Further, upon successful detection of the 

20 gradation areas 310, geometrical information on the game mat 
300, such as the position, orientation and distance thereof, 
can be calculated using a method similar to the one for 
determination of the position and orientation of the game 
card 4 in the virtual three-dimensional coordinate system. 

25 By knowing the position and orientation of the game mat 300 

in advance, the game card 4 provided in the game area 306 can 
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be recognized more efficiently. A description was given with 
reference to Fig. 1 that the frame image captured by the 
imaging apparatus 2 is displayed on the display 7 and a 
character is superimposed on the game card 4 when displayed. 
5 In case the game mat 300 is used, the geometrical information 
on the game mat 300 may be acquired so that a background 
image of, for example, a battle using the game card 4 may be 
superimposed on the game mat 300 in a frame image displayed 
on the display 7. By superimposing a background image on the 

10 game mat 300, which is used for efficient recognition of the 
game card 4, instead of displaying only the game mat 300 on 
the display 7, presentation effects are improved. 
[0105] The gradation areas 310 are also used to 

determine the threshold values of pixel values in the RGB 

15 format used for binarization and extraction of the reference 
cell 100, the corner cells 104 and/or rectangular cells 106 
on the game card 4. For example, it will be assumed the 
corner cells 104 are colored green and the brightness thereof 
is predetermined. In this case, the gradation area 310 may 

20 be colored green and provided with an area where the 

brightness thereof is the same as that of the corner cells 
104. As a result, an area 316 with the same brightness as 
that of the corner cells 104 is located at a predetermined 
position between the center 314 of the gradation area 310 and 

25 the outer edge 312 thereof. Preferably, the area 316 is 

formed concentrically at a middle point between the center 
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314 and the outer edge 312. Accordingly, an area with 
brightness levels higher than that of the area 316 is 
provided between the area 316 and the center 314 in the 
radial direction of the gradation area 310. An area with 
5 brightness levels lower than the area 316 is provided between 
the area 316 and the outer edge 312. 

[0106] It will be assumed that, given the same hue (G) 

and the same brightness (gradation) of the corner cells 104 
and the area 316 in the gradation area 310, and given that 

10 ambient light has no influence, the G pixel value acquired 

via a G filter is 127. The binarization processor 50 sets up 
ranges of RGB pixel values for binarization. If the pixel 
value of the corner cells 104 is known in advance, it is 
ideal that the ranges set up include pixel values close to 

15 the pixel value of the corner cells 104. This is to ensure 
that those pixels with smaller pixel values than that of the 
corner cells 104 (i.e., pixels with higher brightness levels) 
are not extracted as much as possible. Theoretically, the 
binarization processor 50 in this case may set up a range 

20 120-255 of G pixel values, where 120 represents a threshold 
value, for extraction of the corner cells 104. This results 
in images with G pixel values smaller than 120 being coded as 
"0s". By narrowing the range of G pixel values for 
binarization, other noises can be reduced. 

25 [0107] In real world situations, the pixel value of the 

corner cells 104 acquired by the imaging apparatus 2 may 
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differ from a theoretical value (127) due to illumination or 
sunlight entering the imaging apparatus 2. The term 
"theoretical value" is used for the sake of convenience and 
means a pixel "value extracted from image data in a reference 
5 environment. The narrower the range of pixel values set up 
for binarization, the lower the load in subsequent processes. 
Meanwhile, pixel values actually acquired may deviate from 
theoretical values due to influence from ambient light and 
may lie outside a narrowly set range of pixel values for 

10 binarization. The game system 1 according to this example 
addresses this by calibrating the range of pixel values for 
binarization, by using the gradation areas 310. A range of 
pixel values adapted to real-world situations that allows for 
influence from ambient light is set up. 

15 [0108] Fig. 15 shows another structure of the image 

analysis apparatus. The image analysis apparatus 20 is 
provided with a frame image acquirer 40 and a real object 
extractor 42. The real object extractor 42 is provided with 
the binarization processor 50, the area extractor 60 and an 

20 RGB adjuster 62. Unlike the block diagram of Fig. 3, Fig. 15 
shows a structure whereby the range of pixel values for 
binarization is subject to calibration. Processing functions 
of the image analysis apparatus 20 are implemented by a CPU, 
a memory, a program loaded into the memory, etc. Fig. 15 

25 depicts a structure implemented by the cooperation of the 
elements. The program may be built in the image analysis 
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apparatus 20 or supplied from an external source in the form 
of a recording medium. Therefore, it will be obvious to 
those skilled in the art that the functional blocks may be 
implemented by a variety of manners including hardware only, 
5 software only or a combination of both. In the illustrated 
example, the CPU of the image analysis apparatus 20 is 
provided with the functions of the frame image acquirer 40 
and the real object extractor 42. 

[0109] The frame image acquirer 40 acquires a frame 

10 image in a real space captured by the imaging apparatus 2. 
The imaging apparatus 2 acquires frame images at regular 
intervals and, more preferably, generates frame images at 
intervals of 1/60 seconds. The game mat 300 and the game 
card 4 placed on the game mat 300 are located within the 
15 imaging area 5. Therefore, the frame image includes images 
of the game mat 300 and the game card 4. 

[0110] The real object extractor 42 extracts real object 

images from the frame image. Since the purpose here is to 
adjust the range of pixel values in the RGB format for 

20 binarization of the corner cells 104, real object images to 
be extracted are gradation areas 310 on the game mat 300. It 
will be assumed that the corner cells 104 are colored green 
with predefined brightness, and the gradation areas 310 are 
formed as green gradation including the predefined brightness. 

25 [0111] The binarization processor 50 sets up a range of 

RGB values for binarization so as to translate the frame 
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image into a binary bit representation. More specifically, 
in order to extract the gradation areas 310a, 310b and 310c 
with green gradation, the binarization processor 50 sets up a 
range 120-255 of G pixel values, a range 0-20 of R pixel 
5 values and a range 0-20 of B pixel values so as to extract 
pixels in the RGB composite format within these ranges. The 
ranges of R and B pixel values are not set to 0-0 in 
consideration of, for example, the property of a color filter. 
The binarization processor 50 extracts pixels in the 
10 composite format within the ranges of pixel values in the RGB 
format thus established and codes the extracted pixel values 
as binary "Is" . 

[0112] The area extractor 60 retrieves the gradation 

areas 310 on the game mat 300 by referring to the binary data 

15 for the frame image. The gradation of the gradation areas 
310 is such that the G pixel value is smaller at the center 
and the G pixel value is larger at the periphery. As a 
result of binarization, the gradation area 310 is translated 
into a binary bit representation such that pixel values in 

20 the composite format at the center are coded as binary "0s" 

and pixel values in the composite format at the periphery are 
coded as binary "Is", producing a doughnut shape where the 
central area is blank. The area extractor 60 then detects a 
doughnut shape in the binary data. Since there is little 

25 likelihood that features other than the gradation areas 310 
are translated into a doughnut shape, forming the gradation 
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areas 310 to be circular in shape offers significant benefits 
in terms of narrowing down the candidates of the gradation 
areas 310. 

[0113] The four gradation areas 310 are provided at the 

5 vertices of a square- If multiple doughnut areas are 

detected as a result of green-based binarization and black- 
based binarization, those doughnut areas that occur at the 
vertices of the square are extracted and identified as the 
gradation areas 310. The doughnut areas identified as the 
10 green gradation areas 310a, 310b and 310c are communicated to 
RGB adjuster 62 for calibration of threshold values for 
binarization. 

[0114] Figs. 16A-16C show doughnut areas identified as 

gradation areas. A description will now be given of 

15 calibration of the range of G pixel values used for 

binarization and for detection of the corner cells 104 of the 
same color, by referring to the green gradation areas 310a- 
310c. It will be assumed that the threshold of G pixel 
values is 120 and the range of extraction is 120-255. The 

20 RGB adjuster 62 adjusts the range of pixel values in the RGB 

format used for binarization in the binarization processor 50, 
on the basis of binary data for the gradation area 310. 
[0115] Fig. 16A shows a doughnut area having a ratio of 

1:2 (more precisely, 121:256) between the inner diameter and 

25 the outer diameter. In this case, ambient light does not 
exercise any influence in the frame image captured by the 
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imaging apparatus 2. The gradation area 310 is binarized in 
an ideal manner. In this doughnut area, the area 316 with 
the same brightness as that of the corner cells 104 is 
located at a point which divides the center and the outer 
5 edge by a ratio of 1:1 and is included in the black area in 
the doughnut area. Accordingly, by setting the threshold 
value of G pixel values at 120, the corner cells 104 can be 
extracted properly. 

[0116] Fig. 16B shows a state in which the inner 

10 diameter of a doughnut area is too small in comparison with 
the outer diameter thereof. In this case, the area 316 
located at a point which divides between the center and the 
outer edge by a ratio of 1:1 is included in the black area in 
the doughnut area, but an inside area with low gray levels is 
15 also extracted. For example, in the presence of ambient 

light which contains green components, the green light and 
the green color of the gradation area 310 may be superimposed 
on each other, causing the G component of the detected light 
to be intensified. In this case, the pixel values of the 
20 doughnut area as detected are higher than the theoretical 

values of the pixel values in the gradation area 310, causing 
the area coded as binary "Is" to be increased in scale more 
than necessary. 

[0117] This means that, if the binarization processor 50 

25 binarizes an image of the game card 4 by setting up a range 

120-255 of the G pixel values in this case, extraction of the 

SC05022WO0 (SC-70086WO) 



61 

corner cells 104 will be successful, but it also results in 
extracting areas with a lighter green tone than the corner 
cells 104. Since an increase of the number of candidates for 
the corner cells 104 will require extended time to identify 
5 the corner cells 104 , it is preferable that the range of 
pixel values for binarization be set so that the number of 
candidates for the corner cells 104 is reduced. 
[0118] The RGB adjuster 62 holds the distribution of 

shades represented by the gradation of the gradation area 310 

10 and the position of the area in which the same shade as that 
of the corner cells 104 occurs. Therefore, upon receipt of 
the doughnut shape shown in Fig. 16B, the RGB adjuster 62 
determines that the threshold set up (120) is too low and 
determines that the threshold should be set at a higher value. 

15 [0119] For example, the RGB adjuster 62 refers to the 

distribution of shades in the gradation area 310 and 
determines a theoretical value of the G pixel value at that 
point (inner diameter) in the gradation area 310, by 
referring to the distance between the center of the doughnut 

20 area and the internal circle (i.e., the radius of the white 
area) . The G pixel value at that point is lower than 120 
theoretically. In the doughnut area shown in Fig. 16B, 
however, the G pixel value as detected at that point is 120. 
Therefore, the RGB adjuster 62 may increase the threshold by 

25 an amount defined by a difference between the theoretical G 
pixel value in the gradation area 310 and the preset 

SC05022WO0 (SC-70086WO) 



62 

threshold of 120, the increase being necessitated by the 
ambient light. For example, given that the theoretical G 
pixel value in the gradation area 310 occurring at the 
boundary between the white area and the black area in the 
5 doughnut area is 60, a difference of 60 is created with 
respect to the preset threshold value of 120. The RGB 
adjuster 62 then sets up a G pixel threshold value of 180 for 
binarization and communicates the value to the binarization 
unit 50. 

10 [0120] Fig. 16C shows a state in which the inner 

diameter of a doughnut area is larger than half of the outer 
diameter thereof. In this case, the area 316 located at a 
point which divides between the center and the outer edge by 
a ratio of 1:1 is included in the white area in the doughnut 

15 area. This means that the area 316 with the same brightness 
as that of the corner cells 104 is coded as "0s" as a result 
of binarization. For example, in the presence of 
considerably dark ambient light, pixel values lower than the 
theoretical values of the gradation area 310 may be detected. 

20 [0121] This means that, if the binarization processor 50 

binarizes an image of the game card 4 by setting up a range 
120-255 of the G pixel values in this case, extraction of the 
corner cells 104 will not be successful. Therefore, there is 
a need to re-establish the range of G pixel values for 

25 binarization in a manner that a more extensive range of G 
pixel values are extracted so that area 316 with the same 
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brightness as that of the corner cells 104 is properly 
extracted. 

[0122] Upon receipt of the shape of the doughnut area 

shown in Fig. 16C, the RGB adjuster 62 determines that the 
5 threshold (120) set up is too high and determines that the 
threshold should be set at a lower value. For example, the 
RGB adjuster 62 refers to the distribution of shades in the 
gradation area 310 and determines a theoretical value of the 
G pixel value at that point (inner diameter) in the gradation 

10 area 310 by referring to the distance between the center of 

the doughnut area and the inner diameter (i.e., the radius of 
the white area) . The G pixel value at that point is higher 
than 120 theoretically. In the doughnut area shown Fig. 16C, 
however, the G pixel value as detected at that point is 120. 

15 Therefore, the RGB adjuster 62 may lower the threshold by an 
amount defined by a difference between the theoretical G 
pixel value in the gradation area 310 and the preset 
threshold of 120. For example, given that the theoretical G 
pixel value in the gradation area" 310 occurring at the 

20 boundary between the white area and the black area in the 
doughnut area is 180, a difference of 60 is created with 
respect to the preset threshold value of 120. The RGB 
adjuster 62 then sets up a G pixel threshold value of 60 for 
binarization and communicates the value to the binarization 

25 unit 50. 

[0123] Upon receipt of the range of pixel values 
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adjusted by the RGB adjuster 62 , the binarization processor 
50 binarizes the frame image by using the adjusted range. 
The area extractor 60 detects a doughnut area corresponding 
to the gradation area 310 in the binary data and communicates 
5 the detected area to the RGB adjuster 62. RGB adjuster 62 

adjusts the threshold values for binarization by referring to 
the doughnut area and feeds back the adjusted values to the 
binarization processor 50. The above-mentioned calibration 
may continue to be performed until the doughnut shape as 

10 shown in Fig. 16A results. 

[0124] Calibration described above is based on the range 

of G pixel values (threshold) for binarization. Calibration 
based on R pixel values or B pixel values may also proceed 
similarly. While the threshold value for R and B pixel values 

15 for binarization of the green gradation areas 310a, 310b and 
310c is set at 20 by default, the RGB adjuster 62 may 
preferably adjust the default value as required. 
[0125] The gradation areas 310a, 310b and 310c occur at 

three different locations iri a space and, so, the optimal 

20 range of pixel values for binarization may differ depending 
on the location due to influence from ambient light or the 
like. If the direction of exterior lighting is such that the 
gradation areas 310a and 310b are brightly illuminated while 
the gradation area 310c is dark, the threshold value for 

25 binarization may be lowered in adaptation to the gradation 

areas 310c. This will result in successful detection of the 
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corner cells 104 irrespective of the position of the game 
card 4 placed. 

[0126] By lowering the threshold value for binarization, 

however, the likelihood of areas other than the corner cells 
5 104 being extracted is increased in the neighborhood of the 
gradation areas 310a and 310b. In this respect, the optimal 
threshold values for binarization may be determined for the 
respective locations of the gradation areas 310 so that the 
threshold for binarization at an arbitrary position in the 

10 play area 306 may be derived from the threshold values for 

binarization at these three locations. For example, denoting 
the threshold value for binarization in the gradation area 
310b by Tl and the threshold value in the gradation area 310c 
by T2, the threshold value for binarization may be linearly 

15 varied from Tl to T2 on a line' connecting the gradation areas 
310b and 310c. The threshold value for binarization may 
similarly be established for other arbitrary positions. The 
threshold value for binarization at a given point may be 
derived from its distance from the three locations mentioned 

20 above. The above approach ensures successful recognition of 
a code even when the conditions of illumination vary and so 
the code colors imaged vary considerably, depending on the 
card position (e.g., when the card is illuminated from close 
proximity) . 

25 [0127] The calibration according to this example may 

preferably be carried out at regular intervals. In a time- 
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invariant environment , calibration need not be carried out 
frequently. A sensor for detecting variation in the 
environment may be provided so that calibration is carried 
out when the sensor detects a variation. For example, user- 
5 initiated movement of the game mat 300, change in the 

brightness of exterior lighting or the like may represent 
variation in the environment. 

[0128] Calibration may proceed such that the threshold 

value for binarization may be corrected by referring to a 

10 difference between the actual pixel value occurring at a 
predetermined position in the gradation area 310 and the 
measured value obtained by imaging. Alternatively, the 
threshold value for binarization may be corrected so that the 
doughnut area obtained when binarizing the gradation area 310 

15 has a specific shape. In the latter case, the threshold 
value for binarization may be incremented or decremented 
depending on the size of the doughnut area so that the shape 
of the doughnut area approaches a theoretical ideal. Instead 
of using a single threshold value for binarization, an upper 

20 threshold and a lower threshold may be provided so that 
binarization may be carried out by determining whether a 
pixel value falls within the range in between. For example, 
the pixel value may be coded as "0", when it is no less than 
85 and no more than 170, and as "1" otherwise. In this case, 

25 the outer diameter of the gradation area 310 may first be 

determined by binarizing the gradation area 310 depending on 
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whether the pixel value is higher than a threshold value for 
binarization . Measurements may then be taken of the doughnut 
area in the middle identified by the upper and lower 
threshold values which may be corrected so that the doughnut 
5 shape approaches a theoretical ideal. Shades as exhibited on 
the gradation area 310 may be reversed- In this case, the 
shades may be reversed before performing a binarization 
process . 

[0129] As shown in Fig. 1, the imaging apparatus 2 

10 captures an image of the game mat 300 from above and at an 
angle. Thus, the gradation areas 310 may be recognized as 
elliptical even if they are formed as perfect circles. In 
this case, too, it is possible to know the position and 
orientation of the game mat 300 in advance, and so such 

15 information may be used to correct the doughnut shape 

obtained as a result of recognizing the gradation areas 310, 
using affine transformation and similarity transformation, 
before carrying out the calibration described above. 
[0130] Since it is preferable that the game mat 300 be 

20 firmly secured before carrying out calibration, calibration 
may be started only after it is confirmed that the position 
of the game mat 300 remains unchanged subsequent to multiple 
processes for recognition of the gradation areas 310 or only 
after it is. confirmed that the position remains unchanged 

25 subsequent to recognition processes successively carried out 
at a predetermined interval. If the geometrical information 
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of the game mat 300 is obtained as described above, it is 
possible to recognize whether the game card 4 is placed on 
the game mat 300 (i.e., the game card 4 and the game mat 300 
are parallel, and the distance between them is practically 0) , 
5 or the game card 4 is being held by the user (i.e., the game 
card 4 and the game mat 300 are removed from each other or 
are not parallel), by calculating the distance between the 
game mat 300 and the card game 4. 

[0131] Fig. 17 shows a system for transmitting image 

10 data according to the example. The image data transmission 
system 200 includes an image data generator 201, a 
transmitting station 204 for transmitting image data and a 
terminal 202 for receiving the image data transmitted. The 
terminal 202 is provided with a display apparatus 203 for 

15 displaying the received image data. The image data 
transmission system 200 transmits data wirelessly. 
Alternatively, data may be transmitted by wire. 
[0132] The image data generator 201 generates image data 

for the two-dimensional code described above (i.e., the image 

20 data for the two-dimensional code described with reference to 
Figs. 2, 8, 9, 10 and 12) in accordance with a data format 
for display on the display apparatus 203 of the terminal 202. 
The image data for the two-dimensional code shown in Fig. 2 
generated by the image data generator 201 may comprise data 

25 for displaying the multiple rectangular cells at 

predetermined coordinate positions on the display apparatus 
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203 and data for displaying the multiple corner cells at 
coordinate positions that surround the multiple rectangular 
cells. The image data generator 201 configures the data for 
the corner cells and the rectangular cells so that the corner 
5 cells as displayed on the display apparatus 203 appear larger 
than the rectangular cells. 

[0133] The image data for the two-dimensional code shown 

in Fig. 8 generated by the image data generator 201 may 
comprise data for displaying the multiple rectangular cells 

10 at predetermined coordinate positions on the display 

apparatus 203 and data for displaying the multiple corner 
cells at coordinate positions that surround the multiple 
rectangular cells. The image data generator 201 configures 
the data for the corner cells so that the corner cells appear 

15 circular when displayed. 

[0134] The image data for the two-dimensional code shown 

in Fig. 9 generated by the image data generator 201 may 
comprise data for displaying he reference cell having a 
predetermined shape at a predetermined position on the 

20 display, data for displaying the multiple polygonal cells 
within a predetermined area defined with respect to the 
reference cell, and data for displaying the multiple corner 
cells at coordinate positions that surround the area. The 
image data generator configures the data such that at least 

25 one of the reference cell, the polygonal cells and the corner 
cells is assigned a color different from that of the other 
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cells. The image data for the two-dimensional code shown in 
Fig. 10 and 12 generated by the image data generator 201 may 
comprise data for displaying he reference cell having a 
predetermined shape at a predetermined position on the 
5 display, data for displaying the multiple corner cells having 
the same shape, and data for displaying the corner cells 
within predetermined areas. The image data generator 201 
configures the data such that the corner cells are assigned a 
color different from that of the reference cell and the 
10 polygonal cells. 

[0135] The transmitting station 204 transmits the image 

data generated by the image data generator to the terminal 

202. It is preferable that the image data be compressed in a 
format decompressable by the terminal 202. The image data 

15 should also be prepared in a data format displayable by the 
browser function in the terminal 202. The transmitting 
station 204 can reduce the volume of data transmitted by 
transmitting image data for each cell in association with the 
coordinate position thereof when displayed. The terminal 202 

20 displays the two-dimensional code on the display apparatus 

203. In the game system 1 of Fig. 1, the user may use the 
two-dimensional code displayed on the display apparatus 203 
instead of the game card 4 so as to allow the code to be 
imaged by the imaging apparatus 2. With this, the user not 

25 in possession of the game card 4 can participate in the game 
system 1 by downloading the image data to the terminal 202. 
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[0136] Described above is an explanation based on the 

illustrative example. The example of the invention is only 
illustrative in nature and it will be obvious to those 
skilled in the art that variations in constituting elements 
5 and processes are possible and that those variations are 
within the scope of the present invention. 

INDUSTRIAL USABILITY 

[0137] The present invention is applicable to an 

10 articlesuch as a card provided with a two-dimensional code or 
an apparatus for recognizing the two-dimensional code. 
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